Improving edge finite element assembly for geophysical electromagnetic modelling on shared-memory architectures

This work presents a set of node-level optimizations to perform the assembly of edge finite element matrices that arise in 3D geophysical electromagnetic modelling on shared-memory architectures. Firstly, we describe the traditional and sequential assembly approach. Secondly, we depict our vectorized and shared-memory strategy which does not require any low level instructions because it is based on an interpreted programming language, namely, Python. As a result, we obtained a simple parallel-vectorized algorithm whose runtime performance is considerably better than sequential version. The set of optimizations have been included to the work-flow of the Parallel Edge-based Tool for Geophysical Electromagnetic Modelling (PETGEM) which is developed as open-source at the Barcelona Supercomputing Center. Finally, we present numerical results for a set of tests in order to illustrate the performance of our strategy.This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 644202. The research leading to these results has received funding from the European Union's Horizon 2020 Programme (2014-2020) and from Brazilian Ministry of Science, Technology and Innovation through Rede Nacional de Pesquisa (RNP) under the HPC4E Project (www.hpc4e.eu), grant agreement No. 689772. Authors gratefully acknowledge the support from the Mexican National Council for Science and Technology (CONACYT). All numerical tests were performed on the MareNostrum supercomputer of the Barcelona Supercomputing Center - Centro Nacional de Supercomputación (www.bsc.es).Peer ReviewedPostprint (author's final draft

Seismic Wave Simulation for Complex Rheologies on Unstructured Meshes

The possibility of using accurate numerical methods to simulate seismic wavefields on unstructured meshes for complex rheologies is explored. In particular, the Discontinuous Galerkin (DG) finite element method for seismic wave propagation is extended to the rheological types of viscoelasticity, anisotropy and poroelasticity. First is presented the DG method for the elastic isotropic case on tetrahedral unstructured meshes. Then an extension to viscoelastic wave propagation based upon a Generalized Maxwell Body formulation is introduced which allows for quasi-constant attenuation through the whole frequency range. In the following anisotropy is incorporated in the scheme for the most general triclinic case, including an approach to couple its effects with those of viscoelasticity. Finally, poroelasticity is incorporated for both the propagatory high-frequency range and for the diffusive low-frequency range. For all rheology types, high-order convergence is achieved simultaneously in space and time for three-dimensional setups. Applications and convergence tests verify the proper accuracy of the approach. Due to the local character of the DG method and the use of tetrahedral meshes, the presented schemes are ready to be applied for large scale problems of forward wave propagation modeling of seismic waves in setups highly complex both geometrically and physically

Comparison of expansion-based explicit time-integration schemes for acoustic wave propagation

We have developed a von Neumann stability and dispersion analysis of two time-integration techniques in the framework of Fourier pseudospectral (PS) discretizations of the second-order wave equation. The first technique is a rapid expansion method (REM) that uses Chebyshev matrix polynomials to approximate the continuous solution operator of the discrete wave equation. The second technique is a Lax-Wendroff method (LWM) that replaces time derivatives in the Taylor expansion of the solution wavefield with their equivalent spatial PS differentiations. In both time-integration schemes, each expansion term J results in an extra application of the spatial differentiation operator; thus, both methods are similar in terms of their implementation and the freedom to arbitrarily increase accuracy by using more expansion terms. Nevertheless, their limiting Courant-Friedrichs-Lewy stability number S and dispersion inaccuracies behave differently as J varies. We establish the S bounds for both methods in cases of practical use, J≤10, and we confirm the results by numerical simulations. For both schemes, we explore the dispersion dependence on modeling parameters J and S on the wavenumber domain, through a new error metric. This norm weights errors by the source spectrum to adequately measure the accuracy differences. Then, we compare the theoretical computational costs of LWM and REM simulations to attain the same accuracy target by using the efficiency metric J/S. In particular, we find optimal (J,S) pairs that ensure a certain accuracy at a minimal computational cost. We also extend our dispersion analysis to heterogeneous media and find the LWM accuracy to be significantly better for representative J values. Moreover, we perform 2D wave simulations on the SEG/EAGE Salt Model, in which larger REM inaccuracies are clearly observed on waveform comparisons in the range J≤3.C. Spa has received funding from the Chilean Agency CONICYT under the project FONDECYT 11140212, whereas O. Rojas and J. de la Puente have received funding fromthe European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement no.777778 MATHROCKS. The research leading to these results hasreceived funding from the European Union’s Horizon 2020 research and innovation programme under the ChEESE project, grant agreement No. 823844. We also acknowledge funding from the Spanish Ministry Project Geofísica de Altas Prestaciones TIN2016-80957-P.Peer ReviewedPostprint (author's final draft

Application of reduced order modelling in geophysics

The main objective of this work is to effectively design an apriori Reduce Order Method solver to solve parametric high dimensional geophysical problems in a cost effective and fast manner

Parallel 3-D marine controlled-source electromagnetic modelling using high-order tetrahedral Nédélec elements

We present a parallel and high-order Nédélec finite element solution for the marine controlled-source electromagnetic (CSEM) forward problem in 3-D media with isotropic conductivity. Our parallel Python code is implemented on unstructured tetrahedral meshes, which support multiple-scale structures and bathymetry for general marine 3-D CSEM modelling applications. Based on a primary/secondary field approach, we solve the diffusive form of Maxwell’s equations in the low-frequency domain. We investigate the accuracy and performance advantages of our new high-order algorithm against a low-order implementation proposed in our previous work. The numerical precision of our high-order method has been successfully verified by comparisons against previously published results that are relevant in terms of scale and geological properties. A convergence study confirms that high-order polynomials offer a better trade-off between accuracy and computation time. However, the optimum choice of the polynomial order depends on both the input model and the required accuracy as revealed by our tests. Also, we extend our adaptive-meshing strategy to high-order tetrahedral elements. Using adapted meshes to both physical parameters and high-order schemes, we are able to achieve a significant reduction in computational cost without sacrificing accuracy in the modelling. Furthermore, we demonstrate the excellent performance and quasi-linear scaling of our implementation in a state-of-the-art high-performance computing architecture.This project has received funding from the European Union's Horizon 2020 programme under the Marie Sklodowska-Curie grant agreement No. 777778. Furthermore, the research leading to these results has received funding from the European Union's Horizon 2020 programme under the ChEESE Project (https://cheese-coe.eu/ ), grant agreement No. 823844. In addition, the authors would also like to thank the support of the Ministerio de Educación y Ciencia (Spain) under Projects TEC2016-80386-P and TIN2016-80957-P. The authors would like to thank the Editors-in-Chief and to both reviewers, Dr. Martin Cuma and Dr. Raphael Rochlitz, for their valuable comments and suggestions which helped to improve the quality of the manuscript. This work benefited from the valuable suggestions, comments, and proofreading of Dr. Otilio Rojas (BSC). Last but not least, Octavio Castillo-Reyes thanks Natalia Gutierrez (BSC) for her support in CSEM modeling with BSIT.Peer ReviewedPostprint (author's final draft

PETGEM: A parallel code for 3D CSEM forward modeling using edge finite elements

We present the capabilities and results of the Parallel Edge-based Tool for Geophysical Electromagnetic modeling (PETGEM), as well as the physical and numerical foundations upon which it has been developed. PETGEM is an open-source and distributed parallel Python code for fast and highly accurate modeling of 3D marine controlled-source electromagnetic (3D CSEM) problems. We employ the N\'ed\'elec Edge Finite Element Method (EFEM) which offers a good trade-off between accuracy and number of degrees of freedom, while naturally supporting unstructured tetrahedral meshes. We have particularised this new modeling tool to the 3D CSEM problem for infinitesimal point dipoles asumming arbitrarily isotropic media for low-frequencies approximations. In order to avoid source-singularities, PETGEM solves the frequency-domain Maxwell's equations of the secondary electric field, and the primary electric field is calculated analytically for homogeneous background media. We assess the PETGEM accuracy using classical tests with known analytical solutions as well as recent published data of real life geological scenarios. This assessment proves that this new modeling tool reproduces expected accurate solutions in the former tests, and its flexibility on realistic 3D electromagnetic problems. Furthermore, an automatic mesh adaptation strategy for a given frequency and specific source position is presented. We also include a scalability study based on fundamental metrics for high-performance computing (HPC) architectures.Comment: \c{opyright} 2018. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/ This project has received funding from the EC-H2020 under the Marie Sklodowska-Curie grant agreement No. 644202, and from the EC-H2020 under the HPC4E Project, grant agreement No. 68977

Python for HPC geophysical electromagnetic applications: experiences and perspectives

Nowadays, the electromagnetic modelling are a fun-damental tool in geophysics due to their wide field of application: hydrocarbon and mineral exploration, reservoir monitoring, CO storage characterization, geothermal reservoir imaging and many others. In particular, the 3D CSEM forward modelling (FM) is an established tool in the oil & gas industry because of the hope that the application of such methods would eventually lead to the direct detection of hydrocarbons through their insulating properties. Although 3D CSEM FM is nowadays a well-known geophysical prospecting tool and his fundamental mathematical theory is well-established, the state-of-art shows a relative scarsity of robust, flexible, modular and open-source codes to simulate these problems on HPC platforms, which is crucial in the future goal of solving inverse problems. In this talk we describe our experience and perspectives in the development of an HPC python code for the 3D CSEM FM, namely, PETGEM. We focus on three points: 1) 3D CSEM FM theory from a practical point of view, 2) PETGEM features and Python potential for HPC applications, and 3) Modelling results of real-life 3D CSEM FM cases. These points depict that PETGEM could be an attractive and competitive HPC tool to simulate real-scale of 3D CSEM FM in geophysics

HPC and edge elements for geophysical electromagnetic problems: an overview

In Finite Element Methods for solving electromagnetic problems, the use of Nédélec Elements has become very popular. In fact, Nédélec Elements are often said to be a cure to many difficulties that are encountered, particularly eliminating spurious solutions, and are claimed to yield accurate results. In this paper, we present our first steps in developing a Nédélec Elements code for simulation of geophysical electromagnetic problems and first ideas on how implement the key issues of Edge Elements in an efficient way on HPC platforms

HPC geophysical electromagnetics: a synthetic VTI model with complex bathymetry

We introduce a new synthetic marine model for 3D controlled-source electromagnetic method (CSEM) surveys. The proposed model includes relevant features for the electromagnetic geophysical community such as large conductivity contrast with vertical transverse isotropy and a complex bathymetry profile. In this paper, we present the experimental setup and several 3D CSEM simulations in the presence of a resistivity unit denoting a hydrocarbon reservoir. We employ a parallel and high-order vector finite element routine to perform the CSEM simulations. By using tailored meshes, several scenarios are simulated to assess the influence of the reservoir unit presence on the electromagnetic responses. Our numerical assessment confirms that resistivity unit strongly influences the amplitude and phase of the electromagnetic measurements. We investigate the code performance for the solution of fundamental frequencies on high-performance computing architectures. Here, excellent performance ratios are obtained. Our benchmark model and its modeling results are developed under an open-source scheme that promotes easy access to data and reproducible solutions.The work of O.C-R., conducted in the frame of PIXIL project, has been 65% cofinanced by the European Regional Development Fund (ERDF) through the Interreg V-A SpainFrance-Andorra program (POCTEFA2014-2020). BSC authors have received funding from the European Union’s Horizon 2020 programme under the Marie Sklodowska-Curie grant agreement N◦ 777778. Furthermore, the development of PETGEM has received funding from the European Union’s Horizon 2020 programme, grant agreement N◦ 828947, and from the Mexican Department of Energy, CONACYT-SENER Hidrocarburos grant agreement N◦ B-S-69926.Peer ReviewedPostprint (published version